Global uniqueness checking in distributed databases

ABSTRACT

A distributed database processing system for a database composed of data records organized into tables which processes unique index atoms consistently and concurrently. Each attempt to insert a new key value into such a unique index atom in any given node is routed to a unique index atom chairman for processing. The chairman determines whether the request will be granted. If the request is not granted, the requesting unique index atom continues to try to submit the insert. If the request is granted, the requesting unique index atom modifies the content thereof and broadcasts a replicated given unique index atom all other nodes that contain the replicated unique index atom.

BACKGROUND OF THE INVENTION CROSS REFERENCE TO RELATED PATENT

This application claims priority from co-pending U.S. Provisional Pat.Application No. 61/789,671 filed Mar. 15, 2013 for Global UniquenessChecking in Distributed Databases which is incorporated hereby in itsentirety.

U.S. Pat. No. 8,224,860 granted Jul. 17, 2012 for a Database ManagementSystem and assigned to the same assignee as this invention isincorporated in its entirety herein by reference.

FIELD OF THE INVENTION

This invention generally relates to database management systems and morespecifically to a methodology for maintaining unique indexes in adistributed database composed of data records organized into tables.

In many databases unique indexes maintain data integrity by insuringthat no two rows (or records) of data in a table have identical keyvalues. That is, in a unique index an indexed key value can only existin one row or record. An example of such a unique index in a credit carddatabase is the customer’s credit card number. Any index to that creditcard number must assure that a given credit card number is only assignedto one individual; that is, only appears in one row or record of acorresponding logical table. So, steps must be taken to insure that twousers do not attempt to assign the same credit card number to twodifferent individuals; that is, two users do not try to place the sameor different index values in one row. Databases that maintain such afunction are known as being consistent and concurrent. Several methodshave been implemented to assure the consistency and concurrency of suchindexes. A popular method involves quiescing operations so that whileone index is being updated, any other attempt is blocked. This approachhas been implemented in non-shared databases where only a single copy ofthe index exists. Often these methods involved quiescing the entiredatabase.

The above-identified U.S. Pat. No. 8,224,860 discloses a distributeddatabase management system comprising a network of transactional nodesand archival nodes. Archival nodes act as storage managers for all thedata in the database. Each user connects to a transactional node toperform operations on the database by generating queries for processingat that transactional node. A given transactional node need only containthat data and metadata as required to process queries from usersconnected to that node. This distributed database is defined by an arrayof atom classes, such as an index class and atoms where each atomcorresponds to a different instance of the class, such as an index atomfor a specific index. Replications or copies of an atom may reside inmultiple nodes as needed. The atom copy in a given node is processed inthat node.

In this implementation of the above-identified U.S. Pat. No. 8,224,860asynchronous messages transfer among the different nodes to maintaindatabase consistency and concurrency. Specifically, each node in thedatabase network has a unique communication path to every other node.When one node generates a message involving a specific atom, it cancommunicate as necessary with those other nodes that also containreplications of that specific atom. Each node generates its messagesindependently of other nodes. So it is possible that, at any giveninstant, multiple nodes contain replications, or copies, of a given atomand that those different nodes may be at various stages of processingthem. Consequently, operations in different nodes are not synchronized.It is necessary to provide a means for maintaining concurrency andconsistency.

More specifically, in such a database management system, it is possiblefor multiple nodes to generate a message requesting an insert to addspecific information into an index atom for a unique index. If multiplerequests occur at different nodes within a short interval, a racesproblem exists that can produce an erroneous entry in the index atom.Prior methods, such as those involving quiescence, are not readilyapplicable to a distributed database management system of the typediscussed above without introducing unacceptable system performancedegradation. What is needed is a method for handling requested insertsto unique indexes in a distributed database management system.

SUMMARY

Therefore it is an object of this invention to provide a databasemanagement system for a distributed database that processes requestedentries into a unique index in a consistent and concurrent fashion.

Another object of this invention is to provide a database managementsystem for a distributed database that processes requested entries intoa unique index in consistent and concurrent fashion without anysignificant performance degradation.

Yet another object of this invention is to provide a database managementsystem for a distributed database that processes requested entries intoa unique index that eliminates the involvement of nodes that do notinclude that unique index.

In accordance with this invention a unique index is maintained in adistributed database concurrently and consistently. The database iscomposed of data records organized into tables and is distributed over aplurality of interconnected transactional and archival nodes wherein adatabase management system defines a plurality of atom classes fordifferent classes of data and metadata and one of said atom classes isan index class that produces a given index atom for a unique index inthe database and wherein different nodes may include a replication of agiven index atom, one copy of a replicated given index atom beingdesignated a chairman. When another node with a replicated given indexatom, a requesting node, seeks to insert a new entry into its localreplicated given index atom, the requesting node initially inserts theentry into the local replicated given index atom, generates a local-onlyflag and transmits to the chairman a message requesting that the entrybe inserted into the index atom. At the node containing the chairman, itis determined whether the requested entry is unique in the chairman’sreplicated given index atom. If the request is determined to be unique,the chairman accepts the entry and transmits a success message to therequesting node. The requesting node responds by clearing the local-onlyflag and by broadcasting its updated replicated given index atom to allother nodes containing a replicated given index atom whereby the indexatom is maintained consistently and concurrently across all nodes.

BRIEF DESCRIPTION OF THE DRAWINGS

The appended claims particularly point out and distinctly claim thesubject matter of this invention. The various objects, advantages andnovel features of this invention will be more fully apparent from areading of the following detailed description in conjunction with theaccompanying drawings in which like reference numerals refer to likeparts, and in which:

FIG. 1 is a diagram in schematic form of one embodiment of an elastic,scalable, on-demand, distributed data processing system thatincorporates this invention;

FIG. 2 depicts the organization of a transactional node;

FIGS. 3A and 3B depict the logical organization of “atom” objectsgenerated by atom classes shown in FIG. 2 that are useful inimplementing this invention and that might appear at any given time in atransactional node;

FIG. 4 depicts the information in an Index atom that can be involved inthe methodology of this invention;

FIG. 5 depicts the syntax of an exemplary asynchronous message thattransfers between the transactional and archival nodes of the databasesystem of FIG. 1 ;

FIG. 6 depicts messages that are useful in implementing an embodiment ofthis invention; and

FIG. 7 is a flow diagram useful in understanding a method by which arequest for insertion of a key value into a unique index atom occurs.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

A specific database management system in FIG. 1 is an elastic, scalable,on-demand, distributed database system 30 with a plurality of dataprocessing nodes. Nodes N1 through N6 are “transactional nodes” thatprovide user applications access to the database; each of nodes A1 andA2 is an “archival node” and acts as a storage manager to maintain adisk archive of the entire database at each archival node. While anarchival node normally stores the entire database, a singletransactional node contains only that portion of the database itdetermines to be necessary to support transactions being performed atthat transactional node at that time.

Each node in FIG. 1 can communicate directly with each other node in thesystem 30 through a database system network 31. For example, node N1 canestablish a communications path with each of nodes N2 through N6, A1 andA2. Communications between any two nodes is by way of serializedmessages. In one embodiment, the messaging is performed in anasynchronous manner to maximize the bandwidth used by the system therebyto perform various operations in a timely and prompt manner. Typicallythe database system network 31 will operate with a combination ofhigh-bandwidth, low-latency paths (e.g., an Ethernet network) andhigh-bandwidth, high-latency paths (e.g., a WAN network). Each node hasthe capability to restrict use of a low-latency path to time-criticalcommunications (e.g., fetching an atom). The high-latency path can beused for non-critical communications (e.g. a request to updateinformation for a table). Also and preferably, the data processingnetwork of this invention incorporates a messaging protocol, such as theTransmission Control Protocol (TCP) and assures that each node processesmessages in the same sequence in which they were sent to it by othernodes.

FIG. 2 depicts a representative transactional node 32 that links to thedatabase system network 31 and various end users 33. The transactionalnode 32 includes a central processing system (CP) 34 that communicateswith the database system network 31 through a network interface 35 andwith the various users through a user network interface 37. The centralprocessing system 34 also interacts with RAM memory 38 that contains acopy of the database management program that implements this invention.This program functions to provide a remote interface 40, a databaserequest engine 41 and a set 42 of classes or objects. The databaserequest engine 41 only exists on transactional nodes and is theinterface between the high-level input and output commands at the userlevel and system level input and output commands at the system level. Ingeneral terms, its database request engine parses, compiles andoptimizes user queries such as SQL queries into commands that areinterpreted by the various classes or objects in the set 42.

In this system, the classes/objects set 42 is divided into a subset 43of “atom classes,” a subset 44 of “message classes” and a subset 45 of“helper classes.” Additional details of certain of these classes thatare relevant to this invention are described. As will become apparent,at any given time a transactional node only contains those portions ofthe total database that are then relevant to active user applications.Moreover, the various features of this distributed database managementsystem enable all portions of database in use at a given time to beresident in random access memory 38. There is no need for providingsupplementary storage, such as disk storage, at a transactional nodeduring the operation of this system.

Referring to FIG. 3A, a Master Catalog atom 70 tracks the status oftransactional and archival nodes in database system 30 of FIG. 1 . Italso can be considered as an active index that creates and monitors theTransaction Manager atom 71, the Database atom 72, each Schema atom 73,each corresponding set of Table atoms 74 and Table Catalog atoms 75, andSequence ID Managers 82. The Table Catalog atom 75 acts as an activeindex and creates and monitors Index atoms 76, Record States atoms 77,Data atoms 78, Blob States atoms 80 and Blob atoms 81 associated with asingle table. There is one Table Catalog atom 75 for each table.

FIG. 3B is useful in understanding the interaction and management ofdifferent atom types. In this context, neither the Master Catalog atom70 nor the Table Catalog atom 75 performs any management functions. Withrespect to the remaining atoms, the Database atom 72 manages each Schemaatom 73. Each Schema atom 73 manages each related Table atom 74 andSequence ID Manager atom 82. Each Table atom 74 manages itscorresponding Table Catalog atom 75, Index atoms 76, Record States atoms77, Data atoms 78, Blob States atom 80 and Blob atoms81. Still referringto FIG. 3B, the database request engine 41 communicates with the MasterCatalog atom 70, Transaction Manager atom 71, the Database atom 72, eachSchema atom 73, each Table atom 74 and the Sequence ID Managers 82. Thedatabase request engine 41 acts as compiler for a high-level languagesuch as SQL. As a compiler, it parses, compiles and optimizes queriesand obtains metadata and data from atoms for the formation of thevarious fragments of data base information.

Each atom has certain common elements and other elements that arespecific to its type. FIG. 4 depicts an index atom 76 that is thesubject of this invention. Element 76A is a unique identification forthe index atom 76. Pointers 76B and 76C identify a master catalog atomand the creating catalog atom, respectively. Each atom must have achairman that performs functions as described later. Element 76D pointsto the node where the chairman for that atom resides.

Each time a copy of an atom is changed in any transactional node, itreceives a new change number. Element 76E records that change number.Whenever a node requests an atom from another node, there is an intervalduring which time the requesting node will not be known to othertransactional nodes. Element 76F is a list of all the nodes to which thesupplying node must relay messages that contain the atom until therequest is completed.

Operations of the database system are also divided into cycles. A cyclereference element 76G provides the cycle number of the last access tothe atom. Element 76H is a list of the all active nodes that contain theatom. Element 761 includes several status indicators. Elements 76Jcontains a binary tree of index nodes to provide a conventional indexingfunction. Element 76K contains an index level. Such index structures andoperations are known to those in skilled in the art.

As previously indicated, communications between any two nodes is by wayof serialized messages which are transmitted asynchronously using theTCP or another protocol with controls to maintain messaging sequences.FIG. 5 depicts the basic syntax of a typical message 90 that includes avariable-length header 91 and a variable-length body 92. The header 91includes a message identifier code 93 that specifies the message and itsfunction. As this invention envisions a scenario under which differentnodes may operate with different software versions, the header 91 alsoincludes identification 94 of the software version that created themessage. The remaining elements in the header include a localidentification 95 of the sender and information 96 for the destinationof the message and atom identification 97. From this information, arecipient node can de-serialize, decode and process the message.

FIG. 6 depicts four messages that are used in one embodiment of thisinvention. Each time an index node is added to the index, an Index NodeAdded message 150 shown in FIG. 6 is generated that contains an indexkey, record identification and other information for the new index. Whena new index has been fully populated and therefore is ready for use, aTable Index Ready message 151 is generated that can also convert awrite-only index into a readable index. An Insert Request message 160 isgenerated when a non-chairman node seeks to insert a key value into anexisting index atom with a unique index. This message is sent to thechairman. A transmitted Insert Status message 161 updates the status ofthe operation in the non-chairman node as described more fully withrespect to the flow diagram of FIG. 7 .

With this as background, FIG. 7 depicts the process by which areplicated index atom at the node containing the chairman or areplicated index atom at any other node can attempt to insert a keyvalue. This processing is conducted at the node containing the chairmanfor the index atom to receive the key value. If the requesting node isthe chairman, it initiates a “chairman-initiated insert unique index”process 200 in FIG. 7 . If the requesting node is not the chairman, thatnode initiates “non-chairman insert unique index” process 201.

Referring to the process 200, the chairman sets a “local-only” flag instep 202 to indicate that the insert process is underway. The“local-only” flag can be a component of the status states element 761 inFIG. 4 . Still referring to FIG. 7 , the chairman attempts to insertthat key value into its replicated index atom in step 203. If theattempt is not successful, steps 204 and 205 transfer control to step206 to produce a “failure” message 206 and the process 200 terminates,generally with some notice of the failure.

If step 204 determines that the attempt is successful, step 204 controltransfers to step 207. In step 208 the chairman first clears the“local-only” flag associated with the inserted key and then broadcaststhe modified index atom to all other nodes that contain a replication ofthat index atom. More specifically, the chairman transmits an Index NodeAdded message 150 in FIG. 6 to all relevant nodes. When the chairmanbroadcasts a Node Added message to all its peers, the process 200 isdone.

When a non-chairman attempts to insert a new index key value in theprocess 201, step 211 attempts to insert the key value in the index atomand sets a local-only flag associated with the inserted key. If thisattempt fails, step 212 diverts control to step 213 whereupon furtherprocessing terminates and a failure indication is generated. As will beapparent, a failure means that the modified index was in conflict withthe contents of the existing index atom at the requesting node.

If, however, the insert index is entered, step 212 diverts control tostep 214 whereupon the non-chairman attempts to send an Insert Requestmessage, such as the Insert Request message 160 in FIG. 6 , to thechairman that then attempts to insert the entry into its correspondingreplication of the index atom. The insert Request message identifiesitself as the requesting node and contains the index atomidentification, the key value and the proposed table row or record. Thechairman uses step 203 to evaluate the “Insert Request” message anddetermine whether the new key value can be inserted as proposed. If itcan, steps 204 and 206 transfer control to step 215 wherein the chairmanaccepts the modified index and sets a status flag to a “success” state.In step 216 the chairman forms an “Insert Status” message 161, as shownin FIG. 6 , and transmits it to the non-chairman requesting index atom.

In step 220, the requesting non-chairman node processes this InsertStatus message. If the Insert Status message indicates that the chairmanhad accepted the modification to the insert atom, step 221 transferscontrol to step 222 that clears the local-only flag that was set in step211.

If the non-chairman request is not inserted by the chairman in step 203,an Insert Status message is generated with a failed state at step 224and transmitted at step 216 whereupon step 221 diverts to step 223 thatremoves the local-only flag for the specific key value status of theinsert in the requesting node. Then control returns to step 211 torepeat the process. Such a situation may result when the index atom hasbeen updated by a previous request from node N2 in FIG. 1 and the nodeN4 makes a request before processing the broadcast from node N2. Thereturn from step 223 to step 211 will continue until node N4 processesthe message. During the next iteration, steps 212 and 213 will cause thefailure and the process will terminate.

Thus in the case of an insert request by either the chairman or anon-chairman, the chairman is the sole arbiter of whether an index atomis updated with a new key value. In either case, the modified index atomis also replicated to all other nodes containing that index atom. Thus,such an index atom modification occurs consistently and concurrently.

With this understanding, it will be apparent that a database managementsystem for a distributed database that processes requested entries intoa unique index atom in accordance with this invention does so in anorderly fashion so that the all copies of the index atom remains remainin a consistent and concurrent state. This method does not introduce anysignificant performance degradation of those nodes that contain a copyof the unique index atom. Moreover, the process operates without anyinvolvement of nodes that do not include that unique index atom.

This invention has been disclosed in terms of certain embodiments. Itwill be apparent that many modifications can be made to the disclosedapparatus without departing from the invention. Therefore, it is theintent of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of thisinvention.

What is claimed is:
 1. (canceled)
 2. A node in a distributed database,the node comprising: a processor; a network interface to receive arequest to insert a new key value into an index atom, the index atomcomprising a fragment of data and/or metadata associated with an indexof a table included in the distributed database; and a memory, operablycoupled to the processor and storing a local copy of an index atom andinstructions, which, when executed by the processor, cause the node to:determine, in response to the request to insert the new key value intothe index atom, if new key value is unique; in response to determiningthat the new key value is unique, (i) insert the new key value into thelocal copy of the index atom and (ii) broadcast instructions, to othernodes in the distributed database containing copies of the index atom,to insert the new key value into the index atom; and in response todetermining that the new key value is not unique, generate a failuremessage.
 3. The node of claim 2, wherein the request indicates that thenew key value should be inserted into all copies of the index atom inthe distributed database and identifies (i) the index atom, (ii) the newkey value, and (iii) a proposed table row and/or record in thedistributed database.
 4. The node of claim 2, wherein the node isconfigured to determine if the new key value is unique by testing thenew key value for uniqueness against the local copy of the index atom.5. The node of claim 2, wherein the node is configured to determine ifthe new key value by determining that no two rows of data in the tablehave identical key values.
 6. The node of claim 2, wherein the node isfurther configured to generate a local-only flag associated with the newkey value.
 7. The node of claim 6, wherein the node is furtherconfigured to clear the local-only flag associated with the new keyvalue in response to the instructions from another node in thedistributed database.
 8. The node of claim 2, wherein the node isfurther configured to initiate insertion of the new key value into theindex atom.
 9. The node of claim 2, wherein the node is furtherconfigured to remove the new key value from the local copy of the indexatom in response to another node in the distributed database determiningthat the new key value is not unique.
 10. The node of claim 2, whereinthe new key value identifies a unique row in the table.
 11. Adistributed database comprising: nodes comprising respective centralprocessing systems and respective memories to store atoms representingdata stored in the distributed database, the atoms comprising an indexatom comprising a fragment of data and/or metadata associated with anindex of a table included in the distributed database, wherein a subsetof the nodes are configured to store respective copies of the indexatom, and wherein a first node in the subset of the nodes is designatedas a chairman for the index atom and configured to receive a new keyvalue for insertion into a chairman’s copy of the index atom, determinethat the new key value is unique, and broadcast, to each other node inthe subset of nodes, instructions to insert the new key value into theindex atom in response to determining that the new key value is unique,the instructions causing the nodes in the subset of nodes to attempt toinsert the new key value into the respective copies of the index atom.12. The distributed database of claim 11, wherein the nodes are furtherconfigured to communicate with each other via asynchronous messaging.13. The distributed database of claim 11, wherein one node in the subsetof the nodes is configured to transmit a request to the chairman toinsert the new key value into the index atom to the chairman.
 14. Thedistributed database of claim 11, wherein the nodes includetransactional nodes configured to provide user applications access tothe distributed database and archival nodes to maintain respective diskarchives of the distributed database.
 15. The distributed database ofclaim 11, wherein each of the nodes is configured to communicatedirectly with each other node.
 16. The distributed database of claim 11,wherein the index atom includes an element pointing to the chairman. 17.The distributed database of claim 11, wherein the index atom includes alist of the nodes in the subset of the nodes.
 18. The distributeddatabase of claim 11, wherein the chairman is further configured toinsert the new key value into the chairman’s copy of the index atom witha local-only flag before determining that the new key value is uniqueand to clear the local-only flag in response to determining that the newkey value is unique.